Image processing system, image processing integrated circuit, and image processing method

ABSTRACT

A content processing apparatus  200  compresses image data of content in a lossy manner and transmits the image data to a head-mounted display  100  through an interface  300 . An image processing integrated circuit  120  of the head-mounted display  100  transmits the image data to a display panel  122  through an interface  304  whose protocol is different from a protocol of the interface  300 , without decoding and decompressing the image data. The display panel  122  displays the image that has been decoded and decompressed after being transmitted through the interface  304.

TECHNICAL FIELD

The present invention relates to an image processing system, an imageprocessing integrated circuit, and an image processing method thatperform content processing including image display.

BACKGROUND ART

Improvements in information processing techniques and image displaytechniques in recent years have made it possible to experience a videoworld in various forms. For example, a panoramic video is displayed on ahead-mounted display, and an image corresponding to the user's gaze isdisplayed. This makes it possible to increase the sense of immersion inthe video world and improve the operability of an application such as agame. Further, a walk-through system has also been developed that allowsa user wearing a head-mounted display to virtually walk around a spacedisplayed as a video while the user physically moves.

SUMMARY Technical Problems

In order to improve the quality of the image experience described above,higher-resolution and higher-frame-rate images are required. However, itis not easy to achieve both an increase in the size of data to beprocessed and an increase in processing speed. For example, focusing ona video interface between an apparatus that generates and reproduces animage and a display apparatus, expanding the bit width or increasing thebit rate makes it possible to transmit large-size data at high speed.However, this poses a problem of an increase in power consumption andelectromagnetic noise. Further, there is also a limit to routing oftraces and cables in the board.

Further, in general, a data transmission path from image generation toimage display may, in some cases, include a plurality of interfaces withdifferent transfer protocols and electrical standards. For example, in acase where an image generation apparatus is provided separately from ahead-mounted display, it is necessary to secure a communication pathlong enough to cover the user's range of motion. Therefore, it isnecessary to suppress the bit width compared to the data transmissionpath inside the head-mounted display, resulting in the need to transmitdata at a higher bit rate. In such a manner, there is a need to stablytransmit high-resolution image data at a high frame rate whilesuppressing an influence on the power consumption, electromagneticnoise, and the like under the constraints imposed by the characteristicsrequired for each interface.

The present invention has been made in view of the problems describedabove. It is an object of the present invention to provide a techniquethat can stably transmit high-quality image data using a transmissionpath that includes a plurality of interfaces with different protocols.

Solution to Problems

In order to solve the problems described above, one aspect of thepresent invention relates to an image processing system. This imageprocessing system includes a compression section configured to compressdata of an image to be transmitted through a first interface, a relaysection configured to relay transmission of the data of the image fromthe first interface to a second interface with a different protocol, anda display section configured to display the data of the imagetransmitted through the second interface, in which the relay sectionrelays the data of the image without decompressing the data of theimage, and the display section decompresses and displays the data of theimage.

Another aspect of the present invention relates to an image processingintegrated circuit. This image processing integrated circuit includes animage data acquisition section configured to acquire data of an imagethat has been compressed and transmitted through a first interface, anda relay section configured to transmit the data of the image to adisplay panel through a second interface whose protocol is differentfrom a protocol of the first interface, in which the relay sectiontransmits the data of the image without decompressing the data of theimage.

Yet another aspect of the present invention relates to an imageprocessing method. This image processing method includes a step ofcompressing data of an image to be transmitted through a firstinterface, a step of relaying transmission of the data of the image fromthe first interface to a second interface with a different protocol, anda step of displaying the data of the image transmitted through thesecond interface, in which the step of relaying relays the data of theimage without decompressing the data of the image, and the step ofdisplaying decompresses and displays the data of the image.

It is noted that any combinations of the constituent componentsdescribed above and the expressions of the present invention that areconverted between a method, an apparatus, a system, a computer program,data structure, a recording medium, and the like are also effective asaspects of the present invention.

Advantageous Effects of Invention

According to the present invention, high-quality image data can bestably transmitted using a transmission path that includes a pluralityof interfaces with different protocols.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an example of a configuration of acontent processing system according to the present embodiment.

FIG. 2 is a diagram schematically illustrating a data path in thecontent processing system according to the present embodiment.

FIG. 3 is a diagram illustrating a configuration of an internal circuitof a content processing apparatus according to the present embodiment.

FIG. 4 is a diagram illustrating a configuration of functional blocks ofthe content processing apparatus according to the present embodiment.

FIG. 5 is a diagram illustrating a configuration of functional blocks ofa head-mounted display according to the present embodiment.

FIG. 6 is a diagram for describing a change in image data in the contentprocessing apparatus according to the present embodiment.

FIG. 7 is a diagram illustrating an example of the structure of data tobe transmitted by a data input/output section of the content processingapparatus through an interface in the present embodiment.

FIG. 8 is a diagram for describing a flow of image data in thehead-mounted display according to the present embodiment.

FIG. 9 is a diagram illustrating a configuration of functional blocks ofa clock control section according to the present embodiment.

FIG. 10 is a diagram illustrating a configuration of a wirelesscommunication interface between two apparatuses that can be applied tothe present embodiment.

FIG. 11 is a diagram exemplifying an overall configuration of thecontent processing system for a case where the interface illustrated inFIG. 10 is introduced.

FIG. 12 is a diagram exemplifying a flow of processing of compressionand encoding and decoding and decompression that can be implemented in acase where the interface illustrated in FIG. 10 is introduced into thepresent embodiment.

DESCRIPTION OF EMBODIMENT

FIG. 1 illustrates an example of a configuration of a content processingsystem according to the present embodiment. This content processingsystem 1 includes a content processing apparatus 200, a head-mounteddisplay 100, and a flat panel display 302. The content processingapparatus 200 and the head-mounted display 100 are connected to eachother through an interface 300 while the content processing apparatus200 and the flat panel display 302 are connected to each other throughan interface 303. The interfaces 300 and 303 perform wirelesscommunication or connect peripheral devices such as universal serial bus(USB) Type-C.

The content processing apparatus 200 may be further connected to aserver through a network. In this case, the server may provide thecontent processing apparatus 200 with an online application, such as agame, in which a plurality of users can participate through the network.The content processing apparatus 200 basically processes a program ofcontent, generates a display image, and transmits the display image tothe head-mounted display 100 or the flat panel display 302.

In one aspect, the content processing apparatus 200 identifies theposition of the viewpoint and the gaze direction on the basis of theposition and posture of the head of the user wearing the head-mounteddisplay 100, and generates a display image of the corresponding field ofview at a predetermined rate. The head-mounted display 100 and the flatpanel display 302 receive the data of the display image and display thedata as an image of content. In this regard, the purpose of displayingan image is not particularly limited.

For example, the content processing apparatus 200 may generate a virtualworld, which is a stage of a game, as a display image while progressingthe electronic game or may display a still image or a moving image forthe purpose of appreciation or information provision regardless ofwhether the display image depicts a virtual world or a real world.However, the configuration of the content processing system in thepresent embodiment is not limited to the one illustrated in the figure.For example, the content processing apparatus 200 may be connected toeither one of the head-mounted display 100 and the flat panel display302 or may be connected to a plurality of head-mounted displays 100 or aplurality of flat panel displays 302.

Further, the functions of the content processing apparatus 200 may beprovided in a server that is connected to the head-mounted display 100or the flat panel display 302 through the network such as the Internet.Alternatively, the functions of the content processing apparatus 200 maybe provided inside the head-mounted display 100 or the flat paneldisplay 302. It is to be understood by those skilled in the art that theprotocols of the interfaces 300 and 303 may vary as appropriateaccording to these connection modes.

In such a manner, the configuration of the content processing system inthe present embodiment is not particularly limited and is applicable toany configuration in which data is transmitted through a plurality ofinterfaces with different protocols. As a representative example, thefollowing describes a system including the content processing apparatus200, which transmits data including an image of content, and thehead-mounted display 100, which displays the image.

FIG. 2 schematically illustrates a data path in the content processingsystem according to the present embodiment. The head-mounted display 100includes a display panel 122, a speaker 110, and an image processingintegrated circuit 120. The display panel 122 displays an image in frontof the user's eyes. The speaker 110 delivers sound to the user's ears.The image processing integrated circuit 120 is a system-on-a-chip inwhich various function modules including a central processing unit (CPU)are mounted. It is noted that although not illustrated herein, thehead-mounted display 100 may additionally include a single-lens cameraor a multi-lens camera, various motion sensors, and the like. Thesingle-lens camera or a multi-lens camera captures a video in a field ofview corresponding to the orientation of the user's face. Various motionsensors detect the movement of the user's head.

The display panel 122 is a panel having a general display mechanism,such as a liquid-crystal display or an organic electroluminescence (EL)display, and processes an image signal input from the image processingintegrated circuit 120 as appropriate and outputs the image signal as animage. The speaker 110 is a general sound apparatus, such as a headphoneor earphones, which process a sound signal input from the imageprocessing integrated circuit 120 as appropriate and outputs the soundsignal as sound. It is noted that the speaker 110 may have a function ofa microphone that converts a voice uttered by the user into a signal andinputs the signal into the image processing integrated circuit 120.

In such a configuration, consider that data of a high-resolution andhigh-frame-rate image is transmitted from the content processingapparatus 200 to the head-mounted display 100 for display. At this time,the interface 300 between the content processing apparatus 200 and thehead-mounted display 100 needs to support sufficient length that cancover the user's range of motion regardless of whether communication isperformed wiredly or wirelessly. For example, assume a case where thedata is transmitted through a cable. In this case, if the bit width islarge, the cable becomes thick and is difficult to handle.

In order to guarantee the data transmission over a long distance withoutinterfering with the mobility of the user, it is necessary to suppressthe bit width and consequently increase the bit rate in the interface300. Further, the protocol of the interface 300 needs to be capable ofsimultaneously transmitting sound data and the like. Moreover, in orderto secure a distance of at least several meters, transmission with arelatively large power profile is required, taking into account lossesin the transmission path.

On the other hand, an interface 304 between the image processingintegrated circuit 120 and the display panel 122 inside the head-mounteddisplay 100 can be connected over a short distance with a large bitwidth. This makes it possible to suppress the bit rate. Further, sincesound data is handled by an interface 306 between the speaker 110 andthe image processing integrated circuit 120, the protocol of theinterface 304 can be an interface dedicated to image data. Moreover, ingeneral, a short-distance connection using a printed circuit board and aflexible cable enables transfer with a low power profile on theassumption of low losses in the path.

In such a data transmission path, if an attempt is made to support ahigher-resolution and higher-frame-rate image only by expanding the bitwidth or increasing the bit rate, problems such as an increase in powerconsumption and electromagnetic noise and difficulty in routing cablesand traces are likely to become apparent. Moreover, as described above,the allowable values of the bit width and the bit rate vary depending onthe interface. Therefore, it is conceivable to reduce the amount of databy compressing the data to be transmitted, along with the measuresagainst them.

In a case where a lossless compression method is employed here,deterioration of image quality can be avoided. However, since the datasize after compression generally varies depending on the contents of theimage, there is no guarantee that a series of frame data that makes upthe video is always transmitted stably under the same conditions. Forexample, if the data rate after compression exceeds the range that theinterface can handle, there is a possibility that missing frames or thelike may occur. In a case where a lossy compression method is employed,the data size can be suppressed to a predetermined value or lower,thereby stabilizing transmission. However, since the image qualitydeteriorates, repeating the processing of performing decompression andperforming another compression each time the data passes through eachinterface further deteriorates the image quality. Further, the timerequired for the processing of compression and decompression may causedata delays.

Therefore, in the present embodiment, image data is compressed in alossy manner in the content processing apparatus 200, which is atransmission source, to reduce the size of the data to be transmitted asmuch as possible, and the decompression of the data is performed onlyimmediately before the data is displayed on the display panel 122. Thatis, when the data is transmitted between different protocols, the imageprocessing integrated circuit 120 does not decompress or newly compressthe data. This minimizes the deterioration of image quality even withlossy compression and enables high-speed transmission with reduced datasize while achieving both low power consumption and high-quality imagedisplay.

FIG. 3 illustrates a configuration of an internal circuit of the contentprocessing apparatus 200. The content processing apparatus 200 includesa CPU 222, a graphics processing unit (GPU) 224, and a main memory 226.Each of these units is connected to each other through a bus 230. Aninput/output interface 228 is further connected to the bus 230.

A communication section 232, a storage section 234, an output section236, an input section 238, and a recording medium drive section 240 areconnected to the input/output interface 228. The communication section232 includes a peripheral device interface such as a USB or peripheralcomponent interconnect express (PCIe) and a network interface of a wiredor wireless local area network (LAN). The storage section 234 includes ahard disk drive, a nonvolatile memory, and the like. The output section236 outputs data to the head-mounted display 100 and the flat paneldisplay 302. The input section 238 receives data from the head-mounteddisplay 100 and the like. The recording medium drive section 240 drivesa magnetic disk and a removable recording medium such as an optical discor a semiconductor memory.

The CPU 222 controls the entire content processing apparatus 200 byexecuting an operating system stored in the storage section 234. The CPU222 also executes various programs read from a removable recordingmedium and loaded into the main memory 226 or downloaded through thecommunication section 232. The GPU 224 has functions of a geometryengine and a rendering processor, performs rendering processingaccording to rendering instructions received from the CPU 222, andoutputs the result to the output section 236. The main memory 226includes a random access memory (RAM) and stores programs and datanecessary for processing.

FIG. 4 illustrates a configuration of functional blocks of the contentprocessing apparatus 200. The functional blocks illustrated in thisfigure and FIG. 5 to be described later can be implemented as hardwareby configurations such as a CPU, a GPU, a microcontroller, a hard-wiredblock, and various memories, and are implemented as software by programsthat implement various functions such as a data input function, a dataretaining function, an image processing function, and a communicationfunction and are loaded into a memory from a recording medium or thelike. Therefore, it is to be understood by those skilled in the art thatthese functional blocks can be implemented in various forms by hardwareonly, software only, or a combination of hardware and software and arenot limited to any of these forms.

Further, although only the functional blocks used to implement thepresent embodiment are illustrated in this figure and FIG. 5 to bedescribed later, it is to be understood by those skilled in the art thatother functional blocks can be provided in various ways as necessary.The content processing apparatus 200 illustrated in the figure includesa data generation section 20, a data conversion section 24, and a datainput/output section 26. The data generation section 20 generates dataof content to be output. The data conversion section 24 converts thedata of the content into an output format. The data input/output section26 transmits and receives data to and from the head-mounted display 100.

The data generation section 20 generates an image and sound data to bepresented to the user as content. The data generation section 20, forexample, progresses the game in response to the user operations on aninput apparatus, not illustrated, the movement of the user's head, andthe like, and generates a corresponding display image and sound at apredetermined rate. Alternatively, the data generation section 20generates, at a predetermined rate, an image of a region to be displayedin a panoramic image so as to correspond to the field of viewcorresponding to the movement of the user's head. For this purpose, thedata generation section 20 internally retains, for example, model dataof an object to appear in the image, video data to be displayed, and aframe buffer that temporarily stores the image rendered.

The data conversion section 24 performs processing on the data of thecontent generated by the data generation section 20. This processing isnecessary to output the data to the head-mounted display 100. In detail,the data conversion section 24 includes a compression section 28, a dataformation section 30, and an encryption processing section 32. Thecompression section 28 compresses the data of the content by using apredetermined method to reduce the size of the data to be transmitted.In particular, the compression section 28 compresses the data of theimage in a lossy manner so that the data size per unit area aftercompression becomes equal to or less than a certain value regardless ofthe contents of the image.

Here, the compression section 28 compresses data in units of a region ofa predetermined size that is smaller than the entire image of one frameby using a compression method such as entropy encoding or index scalarencoding, so that the head-mounted display 100 can decode and decompressthe data in the same units of the region. Hereinafter, the region usedas a unit when the compression section 28 compresses an image will bereferred to as a “compression unit region.” The size of the compressionunit region is determined through handshaking with the head-mounteddisplay 100 on the basis of a display method or the like of the displaypanel 122.

The data formation section 30 sequentially connects the data of soundand the data of the image in a predetermined format to form output data.Here, as described above, at least the data of the image has beencompressed in a lossy manner on a compression unit region basis by thecompression section 28. Then, the data formation section 30 connects thecompressed data of each compression unit region to each other in rasterorder on the image plane and then in frame order to form stream data. Atthis time, sound data may be further connected so as to synchronize witha blanking period between frames.

The encryption processing section 32 encrypts the data to be output byusing a predetermined algorithm such as a public key system. Datanecessary for encryption such as a public key and the like are acquiredfrom the head-mounted display 100. The data input/output section 26streams and transfers the data of the content that has been convertedinto the output format and then encrypted by the data conversion section24 to the head-mounted display 100 through the interface 300. At thistime, the data input/output section 26 may packetize the data in aformat suitable for the interface 300 before transmitting the data.

Further, the data input/output section 26 also acquires the datanecessary for each processing performed by the data generation section20 and the data conversion section 24 from the head-mounted display 100through the interface 300. Here, “necessary data” may include, forexample, an image captured by a camera included in the head-mounteddisplay, measurement values acquired by various sensors, and otherinformation such as a display method that determines the size of thecompression unit region.

Moreover, the data input/output section 26 transmits informationrelating to a clock and phase that determine a timing at which imagedata is transmitted through the interface 300 to the head-mounteddisplay 100 through the interface 300. For example, the datainput/output section 26 transmits at least one of a pixel clock, ahorizontal synchronization signal (H-sync), a vertical synchronizationsignal (V-sync), and the like. On the basis of this information, thehead-mounted display 100 adjusts a clock that determines a timing atwhich the image is displayed and synchronizes the transmission of thedata from the content processing apparatus 200 with the display in thehead-mounted display 100.

FIG. 5 illustrates a configuration of functional blocks of thehead-mounted display 100. As described above, the head-mounted display100 includes a sensor 112 in addition to the image processing integratedcircuit 120, a display section 123 including the display panel 122, andthe speaker 110. The sensor 112 includes a motion sensor, a camera, andthe like. In the present embodiment, the image processing integratedcircuit 120 sequentially outputs the data of the image transmitted fromthe content processing apparatus 200 to the display panel 122 withoutwaiting for reception of data for one frame. Therefore, even in a casewhere a buffer memory (not illustrated) is provided to temporarily storethe data of the image, its capacity can be smaller than the data size ofone frame.

The image processing integrated circuit 120 includes a data input/outputsection 40, a relay section 42, an image data output section 44, a sounddata output section 46, and a detection value acquisition section 48.The data input/output section 40 transmits and receives data to and fromthe content processing apparatus 200. The relay section 42 performsrelay between interfaces with different protocols. The image data outputsection 44 outputs data of an image out of data of content. The sounddata output section 46 outputs data of sound. The detection valueacquisition section 48 acquires detection values from the sensor 112.

The data input/output section 40 serves as an image data acquisitionsection and acquires the data of the content transmitted from thecontent processing apparatus 200 through the interface 300. At thistime, the data input/output section 40 also acquires informationrelating to a clock used for transmission of image data in the contentprocessing apparatus 200. Further, the data input/output section 40transmits data of a captured image acquired by the sensor 112 andmeasurement values of the motion sensor and the like to the contentprocessing apparatus 200 through the interface 300.

The relay section 42 relays transmission of data between the interface300, which is the interface (first interface) with the contentprocessing apparatus 200, and interfaces 304, 306, and 308, which areinterfaces (second interfaces) with the display section 123, the speaker110, and the sensor 112, respectively, inside the head-mounted display100. Examples of the interface 300, which is the interface with thecontent processing apparatus 200, include USB Type-C, DisplayPort,high-definition multimedia interface (HDMI) 2.1 (registered trademark),and the like.

Examples of the interface 304, which is the interface with the displaysection 123, include a mean interpotential interval (MIPI) displayserial interface (DSI). Examples of the interface 306, which is theinterface with the speaker 110, include inter-IC sound (I2S). Examplesof the interface 308, which is the interface with the sensor 112,include MIPI camera serial interface (CSI), MIPI inter-integratedcircuit (I2C), improved inter-integrated circuit (I3C), and the like.Depending on the system configuration, as described above, the interface300 may be a wide area network such as the Internet or a LAN or may bean on-chip bus inside the image processing integrated circuit 120.

In general, data that has been compressed and encoded at thetransmission source is transmitted through the interface. Subsequently,the data is decoded and decompressed at the transmission destinationfirst, before the data is subjected to further processing or is furthertransmitted. Here, in a case where the data is to be further transmittedthrough another interface, it may be, in some cases, necessary to newlycompress and encode the data because the protocol and payload formatsare different. On the other hand, as described above, the relay section42 according to the present embodiment relays the data of the image,which has been compressed and encoded by the content processingapparatus 200, without decoding and decompressing the data of the image.This avoids the deterioration of image quality due to lossy compressionthat would otherwise be performed multiple times, and, at the same time,minimizes the processing that needs to be performed at the time ofrelaying.

In order to realize smooth relaying of data and image display, thecontent processing apparatus 200 forms transmission data in a formatsuitable for the characteristics of the head-mounted display 100, whichis the data transmission destination. For this purpose, the relaysection 42 determines the format of the transmission data throughhandshaking when, for example, the head-mounted display 100 and thecontent processing apparatus 200 are connected to each other.Specifically, the relay section 42 determines at least one of an imagedisplay method, an image resolution, a frame rate, a pixel format, apixel clock operation frequency, timings of the horizontalsynchronization signal (H-sync) and the vertical synchronization signal(V-sync), the size of the compression unit region, a compression method,the compression data length (compression ratio), a stream dataconfiguration that can be supported by the first and second interfaces,presence or absence of sound data, presence or absence of encryption, apublic key, and the like, which are based on the configurations of thedisplay panel 122 and the relay section 42.

For example, assume a case where the display panel 122 has a displaymethod under which the display panel 122 can be driven per each of unitregions independently. The unit regions are obtained by dividing thescreen. In this case, the content processing apparatus 200 is alsonotified of the size of the unit region (hereinafter referred to as a“display unit region”). In response, the content processing apparatus200 defines the display unit regions or regions obtained by dividing thedisplay unit regions as the compression unit regions and compresses thedata, thereby preventing the data in each compression unit fromstraddling a plurality of display unit regions. Accordingly, individualdisplay unit regions can be independently subjected to displayprocessing.

It is noted that the head-mounted display 100 and the content processingapparatus 200 may also determine the pixel format, that is, the colorspace representing pixels such as RGB, YUV444, YUV422, or YUV420, aformat, color depth, and the like. The relay section 42 includes anencryption processing section 50, a data separation section 52, anoutput order control section 54, a clock control section 56, and a dataanalysis section 58. In a case where the data of the content acquired bythe data input/output section 40 has been encrypted, the encryptionprocessing section 50 decrypts the data. Whether or not to encrypt thedata to be transmitted can be controlled interface by interfaceindependently. Accordingly, the presence or absence of encryption can beoptimized according to the confidentiality each interface inherentlyhas.

For example, data transmission between the content processing apparatus200 and the head-mounted display 100 is inherently less confidentialthan data transmission within the head-mounted display 100. Therefore,the data is encrypted at the time of transmission through the interface300 and is decrypted when the head-mounted display 100 has acquired thedata, so that necessary and sufficient confidentiality can bemaintained. However, there is no particular limitation to whichinterface actually encrypts the data.

The data separation section 52 separates a series of pieces of contentdata into a header, image data, and sound data. In a case whereinformation relating to a clock is acquired simultaneously, thisinformation is also separated. The output order control section 54arranges the pieces of data of the image, which is an array of pieces ofdata in each compression unit, in the order suitable for the displayorder on the display panel 122. At this time, each compressed dataitself is not decoded or decompressed, but is reordered or branched intoa plurality of streams as necessary.

In a case of the display panel 122 under such a method that the displaypanel 122 is driven per display unit region as described above, the datain each compression unit is distributed to the corresponding displayunit region. By making the unit of the compression unit region the sameas the unit of the display unit region or the same as the unit furtherdivided therefrom, the data can be appropriately distributed while thedata remains compressed.

The clock control section 56 adjusts a clock to synchronize thegeneration and output of the content data in the content processingapparatus 200 with the output of the content data in the head-mounteddisplay 100. Specifically, the clock control section 56 detects theclock phase and frequency of the content processing apparatus 200 andadjusts the clock frequency and clock phase inside the head-mounteddisplay 100 to suppress the difference between them to a predeterminedvalue or lower.

Such a configuration enables the head-mounted display 100 to output thesequentially acquired data streams without fail with less timeadjustment. As a result, even in a case where the data of the content istemporarily stored inside the head-mounted display 100, the buffermemory can be of minimum size.

The data analysis section 58 analyzes the data of the content anddetermines control information for controlling the display panel 122 andthe speaker 110 such that the data of the content is output underappropriate conditions. For this purpose, the data analysis section 58may decode and decompress the image data before performing imageanalysis. In a case where a liquid-crystal panel is employed as thedisplay panel 122, a higher resolution can be achieved, but the dynamicrange is insufficient and the response speed is low. In a case where anorganic EL panel is employed, the response speed is high, but it isdifficult to achieve a higher resolution and, moreover, a phenomenoncalled Black Smearing may occur in some cases. Black Smearing is aphenomenon in which color blurring occurs in and around a black region.The data analysis section 58 may make corrections so as to eliminatesuch various adverse effects caused by the display panel.

For example, a dark scene or region in the liquid-crystal panel isidentified through image analysis. In order to improve the dynamic rangein the corresponding frame or region, control information for, forexample, reducing the luminance of the backlight of the liquid-crystalpanel constituting the display panel 122 is determined. For example, ascene or region of intense motion in the liquid-crystal panel isidentified through image analysis. Inserting a black image between thecorresponding frames can reset the liquid crystal and improve theresponse speed.

For example, in a case of the organic EL panel, a black region and aregion around the black region in which Black Smearing is likely tooccur is identified through image analysis. Applying an offset to aluminance value or a gamma value for gamma correction makes colorblurring less outstanding. Additionally, the voltage, color gamut, andthe like that determine the display luminance can also be controlled.The processing based on these pieces of control information may beperformed in any of the display section 123, the speaker 110, and theimage processing integrated circuit 120.

Decoding and decompression and analysis of the image data by the dataanalysis section 58 are performed independently of the transmission ofthe image data to the display section 123. That is, the data of theimage to be displayed on the display panel 122 is relayed in acompressed state, while the image analysis in the data analysis section58 is performed separately. Since the output from the data analysissection 58 is limited to the control information for controlling thedisplay panel 122 and the speaker 110, the transmission path is notoverwhelmed, and the output rates of the image and sound are notaffected. It is noted that the data analysis section 58 may be providedin the display section 123.

The image data output section 44 outputs the compressed data of theimage, the output order of which has been appropriately optimized by theoutput order control section 54, to the display section 123 through theinterface 304. The sound data output section 46 outputs the sound dataseparated by the data separation section 52 to the speaker 110 throughthe interface 306. In a case where the data analysis section 58generates control information for controlling the display panel 122 andthe speaker 110, the image data output section 44 and the sound dataoutput section 46 also transmit such information to the display section123 and the speaker 110, respectively. It is noted that, in order totransmit the control information, the data analysis section 58 may beconnected to the display section 123 and the speaker 110 through adedicated transmission path, not illustrated.

The detection value acquisition section 48 acquires detection valuesfrom the sensor 112 through the interface 308 and outputs the detectionvalues to the relay section 42. Here, the detection values include ameasurement value of the motion sensor, an image captured by the camera,and the like. The relay section 42 transmits these pieces of data to thecontent processing apparatus 200 through the data input/output section40 and the interface 300 as needed.

The display section 123 includes the display panel 122 and a decodingand decompression section 60. The display panel 122 includes a matrix ofdisplay elements for displaying image data and a control section fordriving the matrix of display elements. The decoding and decompressionsection 60 decodes and decompresses image data in the compressed stateacquired from the image processing integrated circuit 120. In a casewhere the display panel 122 has a method under which the display panel122 is driven per predetermined display unit region as described above,the decoding and decompression section 60 may be provided for each ofthe display unit regions independently so that decoding anddecompression can be performed in parallel.

In any case, decoding and decompressing the image data immediatelybefore display can minimize deterioration of image quality even withlossy compression and easily achieve high-speed data transmission at allinterfaces. It is noted that, in a case where the data analysis section58 is provided in the display section 123, the data analysis section 58can analyze the image that has been decoded and decompressed by thedecoding and decompression section 60. On the basis of the controlinformation determined by the data analysis section 58, the displaypanel 122 and the speaker 110 adjust the display and sound outputsettings appropriately.

FIG. 6 is a diagram for describing a change in image data in the contentprocessing apparatus 200. First, the data generation section 20 of thecontent processing apparatus 200 renders an image for one frame in aninternal frame buffer 70. The illustrated frame buffer 70 represents animage plane as a rectangle and stores, for example, data of pixel valuesof the size H in the horizontal direction=3840 pixels and the size V inthe vertical direction=2160 pixels.

The data generation section 20 generates an image for one frame bystoring data of pixels in the frame buffer 70 in raster order in whichscanning from left to right is repeated downward in the first line(Line0), the second line (Line1), . . . , as indicated by the arrow. Thecompression section 28 of the content processing apparatus 200sequentially reads the data of the image from the frame buffer 70 togenerate compressed and encoded image data 72.

At this time, the compression section 28 performs lossy compression on apredetermined compression unit region basis, as described above, so thatwhen decoding and decompression are performed, processing can beperformed in the same units of region independently. The illustratedimage data 72 depicts a block of each compression unit region on theimage plane. In this example, the compression unit regions (e.g.,compression unit regions 74) are regions obtained by dividing the lengthof a region of one line of the image plane into four equal regions. Thatis, the width Ph of the compression unit region=H/4 (pixels) and theheight Pv thereof=1 (pixel). However, the size Ph×Pv of the compressionunit region is not limited and is optimized on the basis of the displaymethod, the resolution, and the frame rate of the display panel 122, thecharacteristics of each interface, the compression method, and the like.

It is noted that the illustrated image data 72 merely depicts an exampleof settings of the compression unit regions on the image plane, and itis not intended to form compressed image data as illustrated in thefigure. That is, in practice, each time the data of pixels constitutingeach compression unit region is stored in the frame buffer 70, thecompression section 28 compresses and encodes the data and sequentiallyoutputs the data. As a result, the data of the image that has beencompressed and encoded is output in raster order of the compression unitregions on the image plane.

FIG. 7 illustrates an example of the structure of data to be transmittedby the data input/output section 26 of the content processing apparatus200 through the interface 300. In the example illustrated in the figure,transmission data 76 has a stream structure in which header information78, compressed data 80 of an image, and a horizontal synchronizationsignal 82 are concatenated per line of the compression unit regions. Inthis figure, each rectangle constituting the compressed data 80represents compressed data in one unit.

Here, the numbers in each rectangle indicate the position coordinates ofthe corresponding compression unit region in the form of (the positionin the horizontal direction, the position in the vertical direction). Inthis example, as illustrated in FIG. 6, the length of a region of oneline of the image plane is assumed to be divided into four equalregions. Therefore, the position coordinate in the horizontal directionis any one of 0, 1, 2, and 3. With lossy compression, the data sizeafter compression of each compression unit region can be controlledregardless of contents of the image. It is noted that, although notillustrated here, the vertical synchronization signal is included in thetransmission data 76 in a vertical blanking interval from the last lineof a frame to the first line of the next frame. Additionally, sound datamay be included in this interval. The data input/output section 26sequentially packetizes a series of transmission data 76 and transmitsthe series of transmission data 76 through the interface 300.

FIG. 8 is a diagram for describing a flow of image data in thehead-mounted display 100. The data of the content having the structureillustrated in FIG. 7 is transmitted from the content processingapparatus 200 to the head-mounted display 100 through the interface 300.The output order control section 54 of the relay section 42 acquires thecompressed data of the image out of the data of the content, andperforms necessary processing to adapt the data to the display section123 and the interface 304 in between. At this time, decoding anddecompression or new compression and encoding of the data are notperformed.

In this example, the display panel 122 is structured such that thedisplay panel 122 can be driven per display unit region independently.As the resolution of the image to be displayed increases, it is becomingmore and more difficult to route a high-speed signal operating on thesame clock to both ends of the screen. Further, the higher theresolution and frame rate become, the faster a drive circuit of thepanel needs to be. Accordingly, display apparatuses in which individualregions obtained by dividing one screen can be driven independently arebecoming more and more popular (refer to, for example, Japanese PatentLaid-open No. Hei 5-80714). For example, driving the pixels of aplurality of display unit regions in parallel can reduce the number ofpixels that need to be scanned per unit time, thereby suppressing thespeed of the drive circuit.

In the example illustrated in the figure, the display panel 122 isstructured such that pixels are driven in raster order, as indicated bythe arrows, per each of display unit regions (e.g., display unit regions92) obtained by vertically dividing the screen into four equal regions.For such a display panel 122, the compression unit region is determinedso as not to straddle adjacent display unit regions. This makes itpossible to distribute the image data in the compressed state to thedrive section of each display unit region. Moreover, the decoding anddecompression processing itself can also be performed in parallel byproviding decoding and decompression sections 60 a, 60 b, 60 c, and 60 dto the respective display unit regions.

That is, if the width of the display unit region is Dh, the width Ph ofthe compression unit region is determined so as to satisfy the followingcondition.

Ph=Dh/n1 (n1 is a natural number)

In this case, the height Pv of the compression unit region may be onepixel or a plurality of pixels. On the other hand, in a case of adisplay panel having display unit regions obtained by horizontallydividing the screen, if the height thereof is Dv, the height Pv of thecompression unit region is determined so as to satisfy the followingcondition.

Pv=Dv/n2 (n2 is a natural number)

In this case, the width Ph of the compression unit region may be onepixel or a plurality of pixels.

In a case of a display panel having display unit regions obtained bydividing the screen both horizontally and vertically, both the width Phand the height Pv of the compression unit region are determined so as tosatisfy the above conditions. The content processing apparatus 200 andthe head-mounted display 100 determine the sizes Ph and Pv of thecompression unit region through prior handshaking on the basis of thesize and the like of the display unit region in the display panel 122.

The output order control section 54 of the head-mounted display 100distributes the data of each compression unit in the transmitted streamdata on the basis of the correspondence between the position on theoriginal image plane and the position of the display unit region on thescreen of the display panel 122. The image data output section 44 thenpacketizes the distributed compressed data as appropriate and transmitseach packet to the display section 123 through the interface 304, whichis individually provided for each display unit region.

Each of the decoding and decompression sections 60 a, 60 b, 60 c, and 60d of the display section 123 decodes and decompresses the acquiredcompressed data in parallel and supplies the result to the drivesection, not illustrated, to drive the display pixels of thecorresponding display unit region. In such a manner, the output ordercontrol section 54 appropriately distributes the image data so that thesubsequent processing can be progressed individually and display can beperformed efficiently.

FIG. 9 illustrates a configuration of functional blocks of the clockcontrol section 56. The clock control section 56 includes a clockgeneration section 98, a phase comparison section 94, and a clockadjustment section 96. The clock generation section 98 is the generationsource of a clock that determines an operation timing in thehead-mounted display 100. The phase comparison section 94 acquires,through the interface 300, data relating to a clock that determines atiming at which image data is transmitted from the content processingapparatus 200, and performs phase comparison with the clock inside thehead-mounted display 100.

The comparison target is at least one of the pixel clock, the horizontalsynchronization signal, and the vertical synchronization signal. In acase where the clock operation frequency differs between the contentprocessing apparatus 200 and the head-mounted display 100, it isdesirable to compare the phase difference of the horizontalsynchronization signal or the vertical synchronization signal. It isnoted that a clock divider, not illustrated, may be incorporated intothe clock control section 56. In this case, a clock divided by the clockdivider may be used by the phase comparison section 94. The clockadjustment section 96 adjusts the timing of the clock generated by theclock generation section 98 on the basis of the result of the comparisonby the phase comparison section 94 and outputs the clock.

That is, the clock inside the head-mounted display 100 is adjusted suchthat the phase difference between the transmission from the contentprocessing apparatus 200 through the interface 300 and the datatransmission inside the head-mounted display 100 becomes a predeterminedvalue or lower. The clock adjusted by the clock adjustment section 96 isalso input into the phase comparison section 94, which compares theclock again with the clock in the interface 300. Then, on the basis ofthe comparison result, the clock adjustment section 96 makes furtheradjustments as necessary. Such feedback control can stabilizesynchronization even in a case of data transmission through differentinterfaces.

Such a configuration can minimize the jitter of the transfer timing thatoccurs between both interfaces. As a result, the data of the imagetransmitted from the content processing apparatus 200 can reach thedisplay without the need for significant time adjustment. This makes itpossible to not only achieve display with slight delay, but also reducethe capacity of the buffer memory that temporarily stores the image datafor time adjustment to smaller than one frame.

According to the present embodiment described above, the system thattransmits and displays image data through the plurality of interfaceswith different protocols includes the relay section that inputs datathat has been compressed and encoded at the transmission source andtransmitted into another interface without decoding and decompressingthe data. This configuration can minimize the deterioration of imagequality even with lossy compression and can significantly suppress thesize of the data to be transmitted even in a case of a high-resolutionand high-frame-rate image. As a result, it is possible to display ahigh-quality image while suppressing the influence on power consumption,electromagnetic noise, the configuration of cables and traces, and thelike.

Further, at the time of initial compression and encoding, the data isgenerated so as to be adapted to the display method or the like of thedisplay panel so that the compressed data can be relayed as it iswithout fail and can reach the display with simple processing. Moreover,in a case where the transmission source and the display destination areoperating on different clocks, the relay section adjusts the clock ofthe display destination to minimize the phase difference. Thisconfiguration can allow the transmitted data to be output withoutstagnation. Accordingly, the capacity of the memory that buffers thedata can be saved while the data can be output with slight delay.

The present invention has been described above on the basis of theembodiment. The embodiment is an exemplification, and it is to beunderstood by those skilled in the art that various modifications can bemade to combinations of individual constituent components and individualprocessing processes in the embodiment and that such modifications alsofall within the scope of the present invention.

For example, in a case where the interface 300 between the contentprocessing apparatus 200 and the head-mounted display 100 performswireless communication, the configuration described below may beintroduced. FIG. 10 illustrates a configuration of a wirelesscommunication interface between the two apparatuses that can be appliedto the present embodiment. An interface 310 includes a pair of awireless conversion section 130 a and a wireless transmission/receptionsection 142 a and a pair of a wireless conversion section 130 b and awireless transmission/reception section 142 b. The wirelesstransmission/reception section 142 a and the wirelesstransmission/reception section 142 b establish wireless communicationtherebetween and transmit and receive data. A conventional technique canbe applied to the protocol of the communication.

The wireless conversion sections 130 a and 130 b perform necessaryconversion processing on the data to be transmitted and receivedwirelessly. In detail, the wireless conversion sections 130 a and 130 brespectively include compression and encoding sections 132 a and 132 b,decoding and decompression sections 134 a and 134 b, datacombination/separation sections 136 a and 136 b, encryption processingsections 138 a and 138 b, and data input/output sections 140 a and 140b. By using a predetermined encoding method, the compression andencoding sections 132 a and 132 b compress and encode the data to betransmitted from the respective corresponding wirelesstransmission/reception sections 142 a and 142 b, that is, the data inputfrom an input A in the figure.

Here, the method of the compression and encoding performed by thecompression and encoding sections 132 a and 132 b may be the same as ordifferent from that of the compression section 28 of the contentprocessing apparatus 200. It is noted that, in a case of wireless datatransfer, the frequency of occurrence of missing data during transferincreases compared to the case of wired data transfer. Therefore, thecompression and encoding sections 132 a and 132 b may employ an errordetection and correction method such as forward error correction (FEC).The decoding and decompression sections 134 a and 134 b decode anddecompress the data received by the respective corresponding wirelesstransmission/reception sections 142 a and 142 b, that is, the data inputfrom an input B in the figure. Here, the decoding and decompressionsections 134 a and 134 b may support both the method of the compressionand encoding performed by the compression section 28 of the contentprocessing apparatus 200 and the method of the compression and encodingperformed by the compression and encoding sections 132 a and 132 b.

In a case where this is applied to the present embodiment, data inputinto the input A of the wireless conversion section 130 a is, forexample, data of content such as an image and sound generated by thecontent processing apparatus 200. Further, data input into the input Aof the wireless conversion section 130 b is, for example, an imagecaptured by the camera included in the head-mounted display 100, data ofsound acquired by the microphone, or the result of an image analysisperformed inside the head-mounted display 100.

The data combination/separation sections 136 a and 136 b combinedifferent types of data to be transmitted from the respectivecorresponding wireless transmission/reception sections 142 a and 142 bor separate data received by the respective corresponding wirelesstransmission/reception sections 142 a and 142 b by type. The encryptionprocessing sections 138 a and 138 b encrypt data to be transmitted fromthe respective corresponding wireless transmission/reception sections142 a and 142 b or decrypt encrypted data received by the wirelesstransmission/reception sections 142 a and 142 b. The data input/outputsections 140 a and 140 b are the interfaces with the respectivecorresponding wireless transmission/reception sections 142 a and 142 b.

FIG. 11 exemplifies an overall configuration of the content processingsystem for a case where the interface 310 illustrated in FIG. 10 isintroduced. A content processing system 2 includes a wireless connectionadapter 150 for an information processing apparatus and a wirelessconnection adapter 152 for a display, in addition to the contentprocessing apparatus 200 and the head-mounted display 100. The wirelessconnection adapter 150 for the information processing apparatus isconnected to the content processing apparatus 200. The wirelessconnection adapter 152 for the display is connected to the head-mounteddisplay 100. The wireless connection adapter 150 for the informationprocessing apparatus includes the wireless conversion section 130 a andthe wireless transmission/reception section 142 a illustrated in FIG.10. The wireless connection adapter 152 for the display includes thewireless conversion section 130 b and the wirelesstransmission/reception section 142 b illustrated in FIG. 10.

The content processing apparatus 200 and the head-mounted display 100may be respectively connected to the wireless connection adapter 150 forthe information processing apparatus and the wireless connection adapter152 for the display wiredly using a standard such as USB Type-C. Forexample, the wireless connection adapter 152 for the display is of astationary type and is installed in the vicinity of a user wearing thehead-mounted display 100. Alternatively, the wireless connection adapter152 for the display may be of a portable type in the shape of, forexample, a backpack, a shoulder bag, a waist bag, or the like andcarried by the user wearing the head-mounted display 100.

Alternatively, the wireless connection adapter 152 for the display maybe shaped so as to allow connector connection with the head-mounteddisplay 100 and be connected to the relay section 42 inside thehead-mounted display 100 by traces on the board. These configurationsallow even the content processing apparatus 200 and the head-mounteddisplay 100 that do not have wireless functions to perform wirelesscommunication. Accordingly, the range of motion of the head-mounteddisplay 100 can be further expanded.

It is noted that the apparatus to which the wireless connection adapter152 for the display is connected is not limited to the head-mounteddisplay but may be a general flat plate display apparatus or the like.For example, as in a case of monitors in the seats of vehicles such astrains, automobiles, and airplanes, in a case where an image outputsource apparatus and a display apparatus are far apart from each otheror there are many display apparatuses, realizing wireless communicationusing the wireless connection adapters as illustrated in the figure cansolve problems relating to cable routing.

Assume a case where the interface 310 is applied to the presentembodiment as illustrated in the figure. In this case, in a case offunctions that overlap between the content processing apparatus 200 andthe head-mounted display 100, either one of these functions may beturned off, or these functions may be used in combination. FIG. 12exemplifies a flow of processing of compression and encoding anddecoding and decompression that can be implemented in a case where theinterface 310 is introduced. In a case of (a), the functions of thecompression and encoding processing and the decoding and decompressionprocessing in the wireless conversion sections 130 a and 130 b areturned off, and the decoding and decompression section 60 of thehead-mounted display 100 decodes and decompresses the data that has beencompressed and encoded by the compression section 28 of the contentprocessing apparatus 200.

In a case of (b), the compression and encoding section 132 a of thewireless conversion section 130 a uses a method (“method b”) forwireless transfer to compress and encode the data that has beencompressed and encoded by the compression section 28 of the contentprocessing apparatus 200 by using another “method a.” That is, in thiscase, the data to be transferred is compressed and encoded twice. Then,the decoding and decompression section 60 of the head-mounted display100 uses the “method a” to decode and decompress the data that has beendecoded and decompressed by the decoding and decompression section 134 bof the wireless conversion section 130 b on the receiving side by usingthe “method b.” In a case of (c), the decoding and decompression section134 a of the wireless conversion section 130 a decodes and decompressesthe data that has been compressed and encoded by the compression section28 of the content processing apparatus 200 by using the “method a,” andthen the compression and encoding section 132 a uses the method (“methodb”) for wireless transfer to compress and encode the data.

Then, the compression and encoding section 132 b of the wirelessconversion section 130 b on the receiving side uses the “method a,”which is the same method as the compression section 28, to compress andencode the data that has been decoded and decompressed by the decodingand decompression section 134 b by using the “method b.” The decodingand decompression section 60 of the head-mounted display 100 decodes anddecompresses the data by using the “method a.” By using one of theseprocessing flows or other combinations, the interface 310 can beintroduced into the present embodiment, and the effects described abovecan be obtained.

INDUSTRIAL APPLICABILITY

In this manner, the present invention is applicable to a contentprocessing apparatus, an image processing apparatus, a game apparatus, adisplay apparatus, a head-mounted display, a content processing systemincluding them, and the like.

REFERENCE SIGNS LIST

-   20: Data generation section-   24: Data conversion section-   26: Data input/output section-   28: Compression section-   30: Data formation section-   32: Encryption processing section-   40: Data input/output section-   42: Relay section-   44: Image data output section-   46: Sound data output section-   48: Detection value acquisition section-   50: Encryption processing section-   52: Data separation section-   54: Output order control section-   56: Clock control section-   58: Data analysis section-   60: Decoding and decompression section-   94: Phase comparison section-   96: Clock adjustment section-   98: Clock generation section-   100: Head-mounted display-   110: Speaker-   112: Sensor-   122: Display panel-   123: Display section-   130 a: Wireless conversion section-   142 a: Wireless transmission/reception section-   150: Wireless connection adapter for information processing    apparatus-   152: Wireless connection adapter for display-   200: Content processing apparatus

1. An image processing system comprising: a compression section configured to compress data of an image to be transmitted through a first interface; a relay section configured to relay transmission of the data of the image from the first interface to a second interface with a different protocol; and a display section configured to display the data of the image transmitted through the second interface, wherein the relay section relays the data of the image without decompressing the data of the image, and the display section decompresses and displays the data of the image.
 2. The image processing system according to claim 1, wherein the compression section compresses the data of the image in units of each of regions determined on a basis of a method of a display panel included in the display section.
 3. The image processing system according to claim 2, wherein the display section drives the display panel for each of display unit regions obtained by dividing a screen of the display panel into a plurality of regions, and the compression section compresses the data of the image in a lossy manner for each of the display unit regions or for each of regions obtained by dividing the display unit regions.
 4. The image processing system according to claim 3, wherein, on a basis of a positional relationship between the display unit regions and the regions used by the compression section as units of compression, the relay section distributes the data of the image to corresponding interfaces among a plurality of the second interfaces each provided for a corresponding one of the display unit regions.
 5. The image processing system according to claim 3, wherein the display section includes a decompression section configured to decompress the data of the image for each of the display unit regions.
 6. The image processing system according to claim 1, wherein the relay section includes a clock control section configured to acquire information relating to a clock that determines a timing at which the data of the image is transmitted through the first interface and adjust a clock that determines a timing at which the image is displayed on the display section.
 7. The image processing system according to claim 1, wherein the relay section outputs the data of the image to the display section without waiting for acquisition of the data of the image for one frame.
 8. The image processing system according to claim 1, wherein the relay section includes an encryption processing section configured to decrypt the data of the image that has been encrypted and acquired through the first interface and input the data of the image into the second interface.
 9. The image processing system according to claim 1, wherein the relay section includes a data analysis section configured to determine control information for controlling a display panel in the display section by analyzing the data of the image and transmit the control information to the display section.
 10. An image processing integrated circuit comprising: an image data acquisition section configured to acquire data of an image that has been compressed and transmitted through a first interface; and a relay section configured to transmit the data of the image to a display panel through a second interface whose protocol is different from a protocol of the first interface, wherein the relay section transmits the data of the image without decompressing the data of the image.
 11. An image processing method performed by an image processing system, the image processing method comprising: a compressing data of an image to be transmitted through a first interface; a relaying transmission of the data of the image from the first interface to a second interface with a different protocol; and a displaying the data of the image transmitted through the second interface, wherein the relaying relays the data of the image without decompressing the data of the image, and the displaying decompresses and displays the data of the image.
 12. A computer program for a computer, comprising: a by an image data acquisition section, acquiring data of an image that has been compressed and transmitted through a first interface; and a by a relay section, transmitting the data of the image to a display panel through a second interface whose protocol is different from a protocol of the first interface, wherein the transmitting transmits the data of the image without decompressing the data of the image. 